Image pickup apparatus, control apparatus, control method, and non-transitory computer-readable storage medium

ABSTRACT

An image pickup apparatus includes an image pickup unit configured to receive an incident light beam from regions of a pupil of an optical system different from each other to output a first image signal and a second image signal, and a calculation unit configured to calculate an evaluation value while relatively shifting the first and second image signals from pixels included in a predetermined range to calculate a defocus amount, and the calculation unit is configured to change a size of the predetermined range so that a center of a visual field range relative to a position of a target pixel does not vary depending on a shift amount between the first and second image signals.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image pickup apparatus whichcalculates a defocus amount by a phase difference detection method usingan image pickup element.

2. Description of the Related Art

Previously, an image pickup apparatus which calculates a defocus amount(difference between a light receiving plane and an imaging plane formedby a lens) by a phase difference detection method (pupil-divided phasedifference detection method) using an image pickup element has beenknown.

Japanese Patent Laid-open No. H07-318793 discloses a method of using ana-sensor array and a b-sensor array which receive incident light beamsfrom different regions in a pupil of an image pickup optical system toobtain a correlation function while relatively shifting object imagesformed on the a-sensor array and the b-sensor array. Japanese PatentLaid-open No. 2008-15754 discloses a method of obtaining a shift amountto give a minimum value for continuous correlation amounts by using amethod of three-point interpolation.

However, in the method disclosed in Japanese patent Laid-open NO.2008-15754, a correct defocus amount cannot be calculated, depending onthe object image (particularly, for three-dimensional object image), byusing the method of three-point interpolation in some cases.

SUMMARY OF THE INVENTION

The present invention provides an image pickup apparatus, a controlapparatus, a control method, and a non-transitory computer-readablestorage medium which are capable of calculating a highly-accuratedefocus amount regardless of an object image when performing focusdetection by a pupil-divided phase difference detection method.

An image pickup apparatus as one aspect of the present inventionincludes an image pickup unit configured to receive an incident lightbeam from regions of a pupil of an optical system different from eachother to output a first image signal and a second image signal, and acalculation unit configured to calculate an evaluation value whilerelatively shifting the first and second image signals from pixelsincluded in a predetermined range to calculate a defocus amount, and thecalculation unit is configured to change a size of the predeterminedrange so that a center of a visual field range relative to a position ofa target pixel does not vary depending on a shift amount between thefirst and second image signals.

A control apparatus as another aspect of the present invention includesan input unit configured to input a first image signal and a secondimage signal output from an image pickup unit based on an incident lightbeam from regions of a pupil of an optical system different from eachother, and a calculation unit configured to calculate an evaluationvalue while relatively shifting the first and second image signals frompixels included in a predetermined range to calculate a defocus amount,and the calculation unit is configured to change a size of thepredetermined range so that a center of a visual field range relative toa position of a target pixel does not vary depending on a shift amountbetween the first and second image signals.

A control method as another aspect of the present invention includes thesteps of acquiring a first image signal and a second image signal outputfrom an image pickup unit based on an incident light beam from regionsof a pupil of an optical system different from each other, andcalculating an evaluation value while relatively shifting the first andsecond image signals from pixels included in a predetermined range tocalculate a defocus amount, and the step of calculating the evaluationvalue includes changing a size of the predetermined range so that acenter of a visual field range relative to a position of a target pixeldoes not vary depending on a shift amount between the first and secondimage signals.

A non-transitory computer-readable storage medium as another aspect ofthe present invention stores a program causing a computer to execute aprocess including the steps of acquiring a first image signal and asecond image signal output from an image pickup unit based on anincident light beam from regions of a pupil of an optical systemdifferent from each other, and calculating an evaluation value whilerelatively shifting the first and second image signals from pixelsincluded in a predetermined range to calculate a defocus amount, and thestep of calculating the evaluation value includes changing a size of thepredetermined range so that a center of a visual field range relative toa position of a target pixel does not vary depending on a shift amountbetween the first and second image signals.

Further features and aspects of the present invention will becomeapparent from the following description of exemplary embodiments withreference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an image pickup apparatus in eachembodiment.

FIG. 2 is a pixel array diagram of an image pickup portion in Embodiment1.

FIG. 3 is a block diagram of an image processor in Embodiment 1.

FIG. 4 is a diagram of explaining a method of calculating a correlationfunction by a defocus amount calculator in Embodiment 1.

FIG. 5 is a diagram of explaining the method of calculating thecorrelation function by the defocus amount calculator in Embodiment 1.

FIG. 6 is a diagram of explaining the method of calculating thecorrelation function by the defocus amount calculator in Embodiment 1.

FIG. 7 is a diagram of explaining the method of calculating thecorrelation function by the defocus amount calculator in Embodiment 1.

FIG. 8 is a pixel array diagram of an image pickup unit in Embodiment 2.

FIG. 9 is a block diagram of an image processor in Embodiment 2.

FIG. 10 is a diagram of explaining a method of calculating a correlationfunction by a defocus amount calculator in Embodiment 2.

FIG. 11 is a diagram of explaining a method of calculating a correlationfunction as a comparative example.

FIGS. 12A and 12B are diagrams of explaining a method of calculating acorrelation amount in which the correlation function is minimized as acomparative example.

FIGS. 13A and 13B are diagrams of explaining the method of calculatingthe correlation amount in which the correlation function is minimized asa comparative example.

FIGS. 14A to 14C are diagrams of explaining a reconstruction process onan image by an image generator in each embodiment.

FIG. 15 is a diagram of explaining the method of calculating thecorrelation function as a comparative example.

FIG. 16 is a diagram of explaining the method of calculating thecorrelation function as a comparative example.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described belowwith reference to the accompanied drawings.

Embodiment 1

First of all, referring to FIG. 1, an image pickup apparatus inEmbodiment 1 of the present invention will be described. FIG. 1 is ablock diagram of an image pickup apparatus 100 (digital camera) in thisembodiment.

A controller 101 is for example a CPU, and it reads an operation programof each block in the image pickup apparatus 100 from a ROM 102 anddevelops the program in a RAM 103 to be executed to control an operationof each block in the image pickup apparatus 100. The ROM 102 is arewritable non-volatile memory, and it stores various information suchas parameters needed to perform the operation of each block, as well asthe operation program of each block in the image pickup apparatus 100.The RAM 103 is a rewritable volatile memory, and it is used as atemporary storage area for data output by the operation of each block inthe image pickup apparatus 100.

An optical system 104 includes lens units and it is an imaging opticalsystem (image pickup optical system) to form an object image on an imagepickup portion 105. In this embodiment, the optical system 104 isprovided as an interchangeable lens removably attached to the imagepickup apparatus 100 (camera body), but it is not limited thereto. Thisembodiment can be applied also to a case in which the optical system 104is integrated with the image pickup apparatus 100 (camera body).

The image pickup portion 105 (image pickup unit) includes an imagepickup element such as a CCD sensor and a CMOS sensor, and it performsphotoelectric conversion on an object image (optical image) formed onthe image pickup element via the optical system 104 to output an analogimage signal to an A/D converter 106. The A/D converter 106 performs A/Dconversion on the analog image signal input from the image pickupportion 105 to output a digital image signal (image data) to the RAM103. The RAM 103 stores the image data output from the A/D converter106.

An image processor 107 performs various image processing, such as whitebalance adjustment, color interpolation, reduction/magnification, andfiltering, on the image data stored in the RAM 103. A recording medium108 is for example a removable memory card, and it records, as arecorded image, the image data processed by the image processor 107 andstored in the RAM 103, the image data output from the A/D converter 106performing the A/D conversion, and the like.

Subsequently, referring to FIG. 2, a pixel array of the image pickupportion 105 (image pickup element) in this embodiment will be described.FIG. 2 is a pixel array diagram of the image pickup portion 105. Theimage pickup portion 105 includes a plurality of pixels 202, and thepixels 202 are regularly arrayed in two dimensions. The pixel 202includes a microlens 201 and a pair of photoelectric conversion portions203 and 204. Hereinafter, in this embodiment, images formed on thephotoelectric conversion portions 203 and 204 are referred to as anA-image and a B-image, respectively.

Subsequently, referring to FIG. 3, image generation and defocus amountcalculation by the image processor 107 will be described. FIG. 3 is ablock diagram of the image processor 107. The image processor 107(controller) includes an image generator 305 (addition unit) and adefocus amount calculator 301 (calculation unit).

The image generator 305 adds a plurality of object images (opticalimages) formed based on light beams passing through regions(pupil-divided regions) in a pupil of the optical system 104 (imagepickup optical system) different from each other to generate a singleobject image formed based on the light beams passing through an entireregion in the pupil of the image pickup optical system. In other words,the image generator 305 adds an A-image signal 306 (A-image) and aB-image signal 307 (B-image) input via an input portion 309 (input unit)from the photoelectric conversion portions 203 and 204, respectively, tooutput an image signal 308 (added signal). The image generator 305 canreconstruct object images on other imaging planes, as well as the objectimage on an image pickup plane (acquisition plane) of the image pickupportion 105. With respect to a reconstruction process of an image, aknown method such as a method disclosed in Japanese Patent Laid-open No.2013-161065 may be applied.

FIGS. 14A to 14C are diagrams of explaining the reconstruction processof the image by the image generator 305, which illustrate reconstructingthe object images on a reconstruction plane 1, an acquisition plane, anda reconstruction plane 2, respectively. When the object image on theacquisition plane of the image pickup portion 105 is to bereconstructed, as illustrated in FIG. 14B, the A-image and the B-imageare added without relatively shifting the images to obtain areconstructed image S_(n) (n is an integer). When the object image onthe imaging plane other than the acquisition plane is to bereconstructed, as illustrated in FIGS. 14A and 14C, the A-image and theB-image are added while relatively shifting the images to obtain thereconstructed image S_(n). As illustrated in FIGS. 14A and 14C, when ashift amount in a shift addition is an odd number, a centroid of thereconstructed image S_(n) is displaced by a half pixel to the left. Theimage processor 107 performs processing such as white balanceadjustment, color interpolation, reduction/magnification, and filteringon the reconstructed image output from the image generator 305. Ifneeded, further processing such as compression and encoding may beperformed on the image processed by the image processor 107, and thenthe processed image is recorded in the recording medium 108 as a storedor recorded image.

In FIG. 3, the defocus amount calculator 301 calculates a defocus amountat a position of a target pixel. In other words, the defocus amountcalculator 301 calculates a defocus amount 304 based on the A-imagesignal 302 (A-image) input from the photoelectric conversion portion 203and the B-image signal 303 (B-image) input from the photoelectricconversion portion 204 via an input portion 310 (input unit) and itoutputs the defocus amount 304. The defocus amount calculator 301calculates a correlation function (correlation amount or correlationvalue) or the like to calculate the defocus amount. A process ofcalculating the correlation function will be described in detail below.With respect to a process of calculating the defocus amount based on thecorrelation function, for example a method disclosed in Japanese PatentLaid-open No. 2008-15754 may be applied.

The defocus amount calculator 301 is capable of generating a defocus mapthat indicates a defocus amount distribution of the object bycalculating the defocus amount while displacing (shifting) the positionof the target pixel by each pixel. The image processor 107 can add abackground blur with a desired size to the recorded image with a deepdepth of field by image processing by referring to the generated defocusmap. With respect to the blur adding process referring to the defocusmap, for example a method disclosed in Japanese Patent Laid-open No.2008-15754 may be applied. The controller 101 specifies the position ofthe target pixel as a pixel position for autofocus detection to receivethe defocus amount, and thus it can use the defocus amount for anautofocusing function to perform focusing while driving a focus lensincluded in the optical system 104 based on the defocus amount.

Next, a correlation function C(S) in this embodiment will be described.The correlation function C(S) is obtained so that the defocus amountcalculator 301 is capable of calculating the defocus amount. First,referring to FIGS. 11 to 13A and 13B, a comparative example of thisembodiment will be described. FIG. 11 is a diagram of explaining amethod of calculating the correlation function C(S) as a comparativeexample, which is basically described in Japanese Patent Laid-open No.H07-318793.

In FIG. 11, an a-sensor array and a b-sensor array are illustrated atthe upper side and the lower side, respectively. Object images obtainedby the a-sensor array and the b-sensor array are indicated as an A-imageand a B-image respectively by gray cells (five-pixel width) withreference to a position of the target pixel (shaded pixel). Thecorrelation function C(S) is calculated while displacing (shifting) theA-image and the B-image relatively depending on a shift amount S (S isan integer, which is within a range of −4≦S≦4 in FIG. 11).

A region combined by the gray cells and white cells is a visual fieldrange related to the position of the target pixel, and a width of thevisual field range broadens with the increase of an absolute value ofthe shift amount S. In FIG. 11, the visual field range is a six-pixelwidth when for example the shift amount S is ±1. On the other hand, thevisual field range is a nine-pixel width when the shift amount S is ±4.Each arrow illustrated in FIG. 11 indicates a center of a visual field,i.e. a center of the visual field range. The correlation function C(S)at the center of the visual field is calculated by the followingexpression (1).C(S)=Σ|a(n+s)−b(n)|  (1)

As illustrated in FIG. 11, when the shift amount S is an even number,the center of the visual field and the position of the target pixelcoincide with each other. On the other hand, when the shift amount S isan odd number, the center of the visual field is displaced (shifted) bya half pixel relative to the position of the target pixel. In thisembodiment, the correlation function C(S) is calculated on conditionthat the shift amount S is an integer. Therefore, a correlation amountSm (correlation value or relative variation amount) at which thecorrelation function C(S) is minimized needs to be obtained withsub-pixel accuracy by using an interpolation process. Then, the defocusamount is calculated based on the correlation amount Sm.

Subsequently, referring to FIGS. 12A and 12B, a method of calculatingthe correlation amount Sm at which the correlation function C(S) isminimized by the using a method of three-point interpolation will bedescribed. FIGS. 12A and 12B are diagrams of explaining the method ofcalculating the correlation amount Sm, and the same method is disclosedin Japanese Patent Laid-open No. 2008-15754. In each of FIGS. 12A and12B, a horizontal axis and a vertical axis indicate the shift amount Sand the correlation function C(S), respectively. FIG. 12A illustratesthe shift amount S in a range of −4≦S≦4, and FIG. 12B illustrates theshift amount S in a range of 0≦S≦4 (enlarged view of part of FIG. 12A).

As illustrated in FIGS. 12A and 12B, the correlation function C(S)(correlation value) is minimized when the shift amount S is 2 (withaccuracy of integers). The correlation amount Sm with sub-pixel accuracyis equal to 2.29 (Sm=2.29) by the interpolation using the expressiondisclosed in Japanese Patent Laid-open No. 2008-15754 according to S=1,2, and 3. Using the correlation amount Sm, the defocus amount can becalculated.

Subsequently, referring to FIGS. 13A and 13B, a case in which a correctinterpolation calculation cannot be performed by using the method ofthree-point interpolation will be described. FIGS. 13A and 13B arediagrams of explaining the method of calculating the correlation amountSm, which illustrate a case in which a correct interpolation calculationcannot be performed. In each of FIGS. 13A and 13B, a horizontal axis anda vertical axis indicate the shift amount S and the correlation functionC(S), respectively. FIG. 13A illustrates the shift amount S in a rangeof −4≦S≦4, and FIG. 13B illustrates the shift amount S in a range of0≦S≦4 (enlarged view of part of FIG. 13A).

As illustrated in FIG. 11, when the shift amount S is an even number,the position of the target pixel and the position of the center of thevisual field coincide with each other. Therefore, the correlationfunction C(S) in this case is calculated at the position of the targetpixel. On the other hand, when the shift amount S is an odd number, theposition of the center of the visual field is displaced by a half pixelto the left relative to the position of the target pixel. Therefore, thecorrelation function C(S) in this case is calculated at a positiondisplaced by a half pixel to the left relative to the position of thetarget pixel.

In the method of calculating the correlation function as a comparativeexample, the correlation function C(S) may be displaced since theposition of the center of the visual field relative to the target pixelvaries depending on the shift amount S, i.e. whether the shift amount Sis either one of an even or odd number. The correlation function C(S) iscalculated by a sigma accumulation represented by expression (1), andaccordingly, a first accumulated value and a final accumulated value inthe sigma accumulation by expression (1) are affected by thedisplacement by a half pixel for the center of the visual field.

When there is no texture at positions of right-end and left-end pixelsfor the A-image and the B-image, the first accumulated value and thefinal accumulated value in the sigma accumulation by expression (1)become small. Therefore, the correlation function C(S) is not displacedregardless of the shift amount S of the even or odd number. As a result,as illustrated in FIGS. 12A and 12B, the correlation function C(S)determined when the shift amount S is the even number and thecorrelation function C(S) determined when the shift amount S is the oddnumber are located on the same curved line.

On the other hand, when there are textures at positions of the right-endand left-end pixels for the A-image and the B-image, the firstaccumulated value and the final accumulated value in the sigmaaccumulation by expression (1) increase. Therefore, the correlationfunction C(S) is displaced depending on whether the shift amount S isthe even or odd number. As a result, as illustrated in FIGS. 13A and13B, the correlation function C(S) determined when the shift amount S isan even number and the correlation function C(S) determined when theshift amount S is an odd number are not located on the same curved line.Accordingly, in the case of FIGS. 13A and 13B, the three-pointinterpolation calculation cannot be precisely performed, and thus thecorrelation amount Sm indicates a value shifted from a correct value(Sm=2.29) illustrated in FIGS. 12A and 12B and it indicates an incorrectvalue (Sm=2.57). In order to solve this problem, this embodimentcalculates the correlation function C(S) by using the following method.

Subsequently, referring to FIGS. 4 and 5, a method of calculating acorrelation function C(S) in this embodiment will be described. FIGS. 4and 5 are diagrams of explaining the correlation function C(S) in thisembodiment. The defocus amount calculator 301 calculates the correlationfunction C(S) by using SAD (Sum of Absolute Difference) represented byexpression (1), and specifically it accumulates difference absolutevalues (absolute differences) between the A-image and the B-image.

In each of FIGS. 4 and 5, an a-sensor array and a b-sensor array areillustrated at the upper side and the lower side, respectively. Objectimages obtained by the a-sensor array and the b-sensor array areindicated as an A-image and a B-image respectively by gray cells(five-pixel width or six-pixel width) with reference to a position ofthe target pixel (shaded pixel). Each arrow illustrated in FIGS. 4 and 5indicates a center of the visual field, i.e. a center of the visualfield range. The number in the gray cell indicates a weightingcoefficient to be applied to the difference absolute value between theA-image and the B-image. In this embodiment, the defocus amountcalculator 301 accumulates the difference absolute values to which theweighting coefficient of 1 or 0.5 is applied while relatively displacingthe A-image and the B-image depending on the shift amount S to calculatethe correlation function C(S).

First, the correlation function C(S) in FIG. 4 will be described. Whenthe shift amount S is an even number, the correlation function C(S) iscalculated similarly to FIG. 11. On the other hand, when the shiftamount S is an odd number, widths of the A-image and the B-image areincreased by a pixel to the right compared to FIG. 11 to be six-pixelwidths. Thus, the center of the visual field is displaced by a halfpixel to the right compared to the case (five-pixel widths) of FIG. 11.Therefore, similarly to the case where the shift amount S is the evennumber, the position of the target pixel and the position of the centerof the visual field coincide with each other (i.e. the position of thecenter of the visual field relative to the position of the target pixelis constant regardless of the shift amount S of the even or odd number).As a result, the correlation function C(S) determined when the shiftamount S is the even number and the correlation function C(S) determinedwhen the shift amount S is the odd number are located on the same curvedline regardless of the shapes of the A-mage and the B-image, andaccordingly the three-point interpolation calculation can be preciselyperformed. In addition, since a sum of the weighting coefficients iskept to be constant (in this embodiment, sum (i.e. five) of the weighingcoefficient (i.e. one) for the five-pixel width is kept) even when thepixel width increases by a pixel, the weighting coefficients for theright-end and left-end pixels are changed (in this embodiment, they areset to 0.5).

Subsequently, the correlation function C(S) in FIG. 5 will be described.When the shift amount S is an odd number, the correlation function C(S)is calculated similarly to FIG. 11. On the other hand, when the shiftamount S is an even number, widths of the A-image and the B-image areincreased by a pixel to the left compared to FIG. 11 to be six-pixelwidths. Thus, the center of the visual field is displaced by a halfpixel to the left compared to the case (five-pixel widths) of FIG. 11.Therefore, similarly to the case where the shift amount S is the oddnumber, the position of the center of the visual field is displaced by ahalf pixel to the left relative to the position of the target pixel(i.e. the position of the center of the visual field relative to theposition of the target pixel is constant regardless of the shift amountS of the even or odd number). As a result, the three-point interpolationcalculation can be precisely performed even in the case of FIG. 5. Inthis embodiment, the weighting coefficients for the right-end andleft-end pixels are set to 0.5.

When the centroid (center of the visual field) of the reconstructedimage generated by the image generator 305 is located at aninteger-pixel position (for example, the reconstructed image S_(n) inFIG. 14B), the defocus amount calculator 301 calculates the correlationfunction C(S) of FIG. 4. On the other hand, when the centroid (center ofthe visual field) of the reconstructed image generated by the imagegenerator 305 is located at a half-pixel position (for example, thereconstructed image S_(n) in each of FIGS. 14A and 14C), the defocusamount calculator 301 calculates the correlation function C(S) of FIG.5. Accordingly, when generating the defocus map, the defocus amountcalculator 301 can align the centroids of the reconstructed image andthe defocus map.

As described above, FIG. 11 and FIGS. 4 and 5 describes the case where asum of the weighting coefficients is five (odd number), but thisembodiment is not limited thereto. Subsequently, referring to FIG. 15and FIGS. 6 and 7, a method of calculating the correlation function C(S)in a case where a sum of the weighting coefficients are four (evennumber) will be described.

FIG. 15 is a diagram of explaining the correlation function C(S) as acomparative example. In FIG. 15, similarly to FIG. 11, the position ofthe center of the visual field is displaced (shifted) by a half pixeldepending on the shift amount S, i.e. whether the shift amount S iseither one of an even or odd number. Therefore, the correlation functionC(S) determined when the shift amount S is an even number and thecorrelation function C(S) determined when the shift amount S is an oddnumber are not located on the same curved line depending on the shapesof the A-image and the B-image, and thus the three-point interpolationcalculation cannot be precisely performed in some cases.

FIGS. 6 and 7 are diagrams of explaining the correlation function C(S)in this embodiment. First, the correlation function C(S) in FIG. 6 willbe described. When the shift amount S is an odd number, the correlationfunction C(S) is calculated similarly to FIG. 15. On the other hand,when the shift amount S is an even number, widths of the A-image and theB-image are increased by a pixel to the left compared to FIG. 15 to befive-pixel widths. In this case, the position of the center of thevisual field is displaced by a half pixel to the left compared to thecase of FIG. 15. Therefore, the center of the visual field determinedwhen the shift amount S is the even number coincides with the center ofthe visual field determined when the shift amount S is the odd number.As a result, the correlation function C(S) determined when the shiftamount S is the even number and the correlation function C(S) determinedwhen the shift amount S is the odd number are located on the same curvedline regardless of the shapes of the A-image and the B-image, andtherefore the three-point interpolation calculation can be preciselyperformed. Since a sum of the weighting coefficients is kept to beconstant (in this embodiment, the sum is four) even when the pixel widthincreases by a pixel, the weighting coefficients for the right-end andleft-end pixels are set to 0.5.

Subsequently, the correlation function C(S) in FIG. 7 will be described.When the shift amount S is an even number, the correlation function C(S)is calculated similarly to FIG. 15. On the other hand, when the shiftamount S is an odd number, widths of the A-image and the B-image areincreased by a pixel to the right compared to FIG. 15 to be five-pixelwidths. In this case, the position of the center of the visual field isdisplaced by a half pixel to the right compared to the case of FIG. 15.Therefore, the center of the visual field determined when the shiftamount S is the even number coincides with the center of the visualfield determined when the shift amount S is the odd number. As a result,the three-point interpolation calculation can be precisely performedeven in this case. The weighting coefficients for the right-end andleft-end pixels are set to 0.5.

When the centroid (center of the visual field) of the reconstructedimage generated by the image generator 305 is located at aninteger-pixel position (for example, the reconstructed image S_(n) inFIG. 14B), the defocus amount calculator 301 calculates the correlationfunction C(S) of FIG. 6. On the other hand, when the centroid (center ofthe visual field) of the reconstructed image generated by the imagegenerator 305 is located at a half-pixel position (for example, thereconstructed image S_(n) in each of FIGS. 14A and 14C), the defocusamount calculator 301 calculates the correlation function C(S) of FIG.7. Accordingly, when generating the defocus map, the defocus amountcalculator 301 can align the centroids of the reconstructed image andthe defocus map.

In this embodiment, the correlation function C(S) is calculated by usingthe SAD represented by expression (1). However, this embodiment is notlimited thereto, and alternatively the NCC (Normalized CrossCorrelation) represented by the following expressions (2) and (3) may beused.C(S)=Σa(n+s)×b 9 n)/δ  (2)δ(S)=(√Σa(n+s)²)×(√Σb(n)²)  (3)

While the SAD decreases with the increase of the correlation, the NCCincreases with the increase of the correlation. Therefore, when themethod of three-point interpolation is applied to the NCC, it isnecessary to set the value to decrease with the increase of thecorrelation by inversing a sign of the correlation function C(S) inadvance or the like. Correlation expressions other than the SAD and theNCC may be used if a degree of coincidence for a pair of object imagescan be calculated.

In this embodiment, the correlation amount Sm (relative variationamount) is obtained by using the correlation function C(S). However,this embodiment is not limited thereto, and alternatively thecorrelation amount Sm may be obtained by using an image-qualityevaluation value disclosed in Japanese Patent Laid-open No. 2013-235047or the like. The image-quality evaluation value disclosed in JapanesePatent Laid-open No. 2013-235047 is P(S) represented by the followingexpression (5).F(n,S)=a(n+s)+b(n)  (4)P(S)=Σ|−F(n−1,S)+2×F(n,S)−F(n+1,S)|  (5)

In expression (4), symbol F(n,S) denotes a reconstructed image on theimage pickup plane of the image pickup portion 105 and other imagingplanes, and it is obtained by adding the A-image and the B-image whileshifting them by the shift amount S. Since the image-quality evaluationvalue P(S) is, as represented by expression (5), an integrated value ofthe amplitude determined after applying a high-pass filter to thereconstructed image F(n,S), it corresponds to an evaluation value of afocusing degree of the reconstructed image F (n,S). An imaging plane onwhich the focusing degree is maximized is the imaging plane of theoptical system 104, and a difference between the imaging plane of theoptical system 104 and the image pickup plane (light receiving plane) ofthe image pickup portion 105 is the defocus amount. Therefore, thedefocus amount calculator 301 is capable of calculating the defocusamount by obtaining the correlation amount Sm at which the image-qualityevaluation value P(S) is maximized. The image-quality evaluation valueP(S) increases with the increase of the focusing degree. Therefore, whenapplying the method of three-point interpolation to the image-qualityevaluation value, it is necessary to set the value to decrease with theincrease of the focusing degree by inversing the sign of theimage-quality evaluation value P(S) in advance. Focusing degreeexpressions other than the image-quality evaluation value may be used ifthe focusing degree of the reconstructed image can be calculated.

According to this embodiment, an image pickup apparatus can be providedwhich is capable of performing correct three-point interpolationcalculation regardless of an object image when detecting a defocusamount by a pupil-divided phase difference detection method for dividinga pupil into two regions horizontally.

Embodiment 2

Next, an image pickup apparatus in Embodiment 2 of the present inventionwill be described. A basic configuration of the image pickup apparatusin this embodiment is the same as that of the image pickup apparatus 100in Embodiment 1 described referring to FIG. 1.

Referring to FIG. 8, a pixel array of an image pickup portion 105 a(image pickup element) in this embodiment will be described. FIG. 8 is apixel array diagram of the image pickup portion 105 a. The image pickupportion 105 a includes a plurality of pixels 806, and the pixels 806 areregularly arrayed in two dimensions. Each of the pixels 806 includes amicrolens 805 and two pairs of photoelectric conversion portions 801 and804 and photoelectric conversion portions 802 and 803. Hereinafter, inthis embodiment, images formed by the photoelectric conversion portions801, 802, 803, and 804 are referred to as an A-image, a B-image, aC-image, and a D-image, respectively.

Subsequently, referring to FIG. 9, image generation and defocus amountcalculation by an image processor 107 a will be described. FIG. 9 is ablock diagram of the image processor 107 a. The image processor 107 a(controller) includes an image generator 907 (addition unit) and adefocus amount calculator 901 (calculation unit).

The image generator 907 adds a plurality of object images (opticalimages) formed based on light beams passing through regions(pupil-divided regions) in a pupil of the optical system 104 (imagepickup optical system) different from each other to generate a singleobject image formed based on the light beams passing through an entireregion in the pupil of the image pickup optical system. In other words,the image generator 907 adds an A-image signal 908 (A-image), a B-imagesignal 909 (B-image), a C-image signal 910 (C-image), and a D-imagesignal 911 (D-image) input via an input portion 913 (input unit) fromthe photoelectric conversion portions 801, 802, 803, and 804,respectively. Then, the image generator 907 outputs an image signal 912(added signal) generated by adding each of the image signals. The imagegenerator 907 reconstructs an object image on an image pickup plane(acquisition plane) of the image pickup portion 105. The image processor107 a performs processing such as white balance adjustment, colorinterpolation, reduction/magnification, and filtering on thereconstructed image output from the image generator 907. The imageprocessed by the image processor 107 a is recorded in the recordingmedium 108 as a recorded image.

In FIG. 9, the defocus amount calculator 901 calculates a defocus amountat a position of a target pixel. In other words, the defocus amountcalculator 901 inputs an A-image signal 902 (A-image), a B-image signal903 (B-image), a C-image signal 904 (C-image), and a D-image signal 905(D-image) input via an input portion 914 (input unit) from thephotoelectric conversion portions 801, 802, 803, and 804, respectively.Then, the defocus amount calculator 901 calculates a defocus amount 906based on each of the input image signals and outputs the defocus amount906. The defocus amount calculator 901 calculates a correlation function(correlation amount or correlation value) or the like to calculate thedefocus amount.

The defocus amount calculator 901 is capable of generating a defocus mapthat indicates a defocus amount distribution of the object bycalculating the defocus amount while displacing (shifting) the positionof the target pixel by each pixel. The image processor 107 a (defocusamount calculator 901) can add a background blur with a desired size tothe recorded image with a deep depth of field by image processing byreferring to the generated defocus map. The defocus amount calculator901 (controller 101) specifies the position of the target pixel as apixel position for autofocus detection to calculate the defocus amount,and thus it can use the calculated defocus amount for the autofocusdetection.

Subsequently, a correlation function C(S) in this embodiment will bedescribed. First, referring to FIG. 16, a comparative example of thisembodiment will be described. FIG. 16 is a diagram of explaining amethod of calculating the correlation function C(S) as a comparativeexample.

In FIG. 16, an a-sensor group, a b-sensor group, a c-sensor group, and ad-sensor group are illustrated at the upper left, upper right, lowerleft, and lower right, respectively. Object images obtained by thea-sensor group, the b-sensor group, the c-sensor group, and the d-sensorgroup are indicated as an A-image, a B-image, a C-image, and a D-imagerespectively by gray cells (5×5 pixels) with reference to a position ofthe target pixel (shaded pixel). The correlation function C(S) iscalculated while displacing (shifting) the A-image, the B-image, theC-image, and the D-image relatively depending on a shift amount S (S isequal to 0, −1, or −2 in FIG. 16).

A region combined by the gray cells and white cells is a visual fieldrange related to the position of the target pixel, and a width of thevisual field range broadens with the increase of an absolute value ofthe shift amount S. Each arrow illustrated in FIG. 16 indicates a centerof a visual field (horizontal center of a visual field or verticalcenter of a visual field), i.e. a center of the visual field range.Applying the correlation function C(S) represented by expression (1) tothe image pickup portion 105 a (image pickup element) in FIG. 8, thecorrelation function C(S) at the center of the visual field in thisembodiment is represented by the following expression (6).C(S)=Σ|a(n+s,n+s)−d(n,n)|+Σ|b(n,n+s)−c(n+s,n)|  (6)

In the method of FIG. 16 as a comparative example, when the shift amountS is an even number, the center of the visual field and the position ofthe target pixel coincide with each other. On the other hand, when theshift amount S is an odd number, the center of the visual field isdisplaced (shifted) by a half pixel relative to the position of thetarget pixel. In other words, similarly to FIG. 11, the center of thevisual field is displaced by a half pixel depending on the shift amountS, i.e. whether the shift amount S is either one of an even or oddnumber. Therefore, the correlation function C(S) determined when theshift amount S is an even number and the correlation function C(S)determined when the shift amount S is an odd number are not located onthe same curved line depending on the shapes of the A-image, theB-image, the C-image, and the D-image, and thus the three-pointinterpolation calculation cannot be precisely performed in some cases.

Subsequently, referring to FIG. 10, a method of calculating thecorrelation function C(S) in this embodiment will be described. FIG. 10is a diagram of explaining the correlation function C(S) in thisembodiment. In FIG. 10 (this embodiment), when the shift amount S is aneven number, the correlation function C(S) is calculated similarly toFIG. 16 (comparative example). On the other hand, when the shift amountS is an odd number, the sizes (widths) of the A-image, the B-image, theC-image, and the D-image increase by a pixel to the lower side and tothe right side, compared to FIG. 16, to be 6×6 pixels. Accordingly, thecenter of the visual field is displaced by a half pixel to the lowerside and the right side compared to FIG. 16 (five-pixel widths).Therefore, similarly to the case in which the shift amount S is the evennumber, the position of the target pixel and the position of the centerof the visual field coincide with each other (i.e. the position of thecenter of the visual field relative to the position of the target pixelis constant regardless of the shift amount S of the even or odd number).As a result, the correlation function C(S) determined when the shiftamount S is the even number and the correlation function C(S) determinedwhen the shift amount S is the odd number are located on the same curvedline regardless of the shapes of the A-image, the B-image, the C-image,and the D-image, and therefore the three-point interpolation calculationcan be precisely performed. Since a sum of the weighting coefficients iskept to be constant (in this embodiment, 25 as a sum of the weightingcoefficient 1 for 5×5 pixels is kept) even when the pixel size (pixelwidths) to the lower side and to the right side increases by a pixel,the weighting coefficients for peripheral pixels are changed. In thisembodiment, as illustrated in FIG. 10, the weighting coefficient forfour corner pixels is set to 0.25 and the weighting coefficient forperipheral pixels except the four corner pixels is set to 0.5.

According to this embodiment, an image pickup apparatus can be providedwhich is capable of performing correct three-point interpolationcalculation regardless of an object image when detecting a defocusamount by a pupil-divided phase difference detection method for dividinga pupil into four regions horizontally.

As described above, in each embodiment, the image pickup unit (imagepickup portion 105) receives an incident light beam from regions(pupil-divided regions) of a pupil of an optical system 104 differentfrom each other to output a first image signal (A-image signal) and asecond image signal (B-image signal). A calculation unit (defocus amountcalculator 301 or 901 of an image processor 107) calculates anevaluation value (defocus evaluation value) while relatively shiftingthe first and second image signals (phases of the image signals) frompixels included in a predetermined range to calculate a defocus amount.Then, the calculation unit changes a size (number of reference pixels)of the predetermined range so that a center of a visual field range(center of the visual field) relative to a position of a target pixeldoes not vary depending on a shift amount between the first and secondimage signals. In other words, the calculation unit changes the size ofthe predetermined range so that a position of the center of the visualfield is fixed even in any shift amount. The predetermined range means arange of pixels in each sensor array corresponding to an object image,and for example is a range illustrated by gray cells in each of FIGS. 4to 7 and 10.

Preferably, the center of the visual field range coincides with a centerof the target pixel or is displaced by a half pixel from a center of thetarget pixel. Preferably, the calculation unit sets the number of thepixels included in the predetermined range to an even number when theshift amount is a first shift amount. On the other hand, the calculationunit sets the number of the pixels included in the predetermined rangeto an odd number when the shift amount is a second shift amount.

Preferably, an image pickup apparatus 100 further includes an additionunit (image generator 305 or 907) which adds the first and second imagesignals. More preferably, the addition unit adds the first and secondimage signals shifted by a third shift amount from each other (acquiresa reconstructed image). Then, the calculation unit shifts the first andsecond image signals by a fourth shift amount to calculate theevaluation value, and changes the size of the predetermined rangedepending on the third shift amount (shift amount for acquiring thereconstructed image) and the fourth shift amounts (shift amount forcalculating the evaluation value).

Preferably, the calculation unit is capable of calculating a firstevaluation value calculated by using a first weighting coefficient (forexample, 1) for all pixels in the predetermined range. In addition, thecalculation unit is capable of calculating a second evaluation valuecalculated by using a second weighting coefficient (for example, 0.5)smaller than the first weighting coefficient for part of the pixels (forexample, pixel at an end of the predetermined range) in thepredetermined range. Then, the calculation unit determines, depending onthe size of the predetermined range, either one of the first or secondevaluation value to be calculated.

Preferably, the evaluation value is information (such as a correlationvalue) related to a degree of coincidence between the first and secondimage signals. Preferably, the evaluation value is information (such asa contrast evaluation value) related to a focusing degree of the firstand second image signals. Preferably, the calculation unit performsthree-point interpolation calculation based on the evaluation value tocalculate the defocus amount.

According to each embodiment, an image pickup apparatus, a controlapparatus, a control method, and a non-transitory computer-readablestorage medium can be provided which are capable of calculating adefocus amount with high accuracy by using a method of three-pointinterpolation regardless of an object image when performing focusdetection by a pupil-divided phase difference detection method.

Other Embodiments

Embodiment (s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2014-129161, filed on Jun. 24, 2014, which is hereby incorporated byreference wherein in its entirety.

What is claimed is:
 1. An image pickup apparatus comprising: an imagepickup unit configured to receive an incident light beam from regions ofa pupil of an optical system different from each other to output a firstimage signal and a second image signal; and a calculation unitconfigured to calculate an evaluation value while relatively shiftingthe first and second image signals from pixels included in apredetermined range to calculate a defocus amount, wherein thecalculation unit is configured to change a size of the predeterminedrange so that a center of a visual field range relative to a position ofa target pixel does not vary depending on a shift amount between thefirst and second image signals.
 2. The image pickup apparatus accordingto claim 1, wherein the center of the visual field range coincides witha center of the target pixel.
 3. The image pickup apparatus according toclaim 1, wherein the center of the visual field range is displaced by ahalf pixel from a center of the target pixel.
 4. The image pickupapparatus according to claim 1, wherein the calculation unit isconfigured to: set the number of the pixels included in thepredetermined range to an even number when the shift amount is a firstshift amount, and set the number of the pixels included in thepredetermined range to an odd number when the shift amount is a secondshift amount.
 5. The image pickup apparatus according to claim 1,further comprising an addition unit configured to add the first andsecond image signals.
 6. The image pickup apparatus according to claim5, wherein the addition unit is configured to add the first and secondimage signals shifted by a third shift amount from each other, andwherein the calculation unit is configured to: shift the first andsecond image signals by a fourth shift amount to calculate theevaluation value, and change the size of the predetermined rangedepending on the third and fourth shift amounts.
 7. The image pickupapparatus according to claim 1, wherein the calculation unit is capableof calculating a first evaluation value calculated by using a firstweighting coefficient for all pixels in the predetermined range and asecond evaluation value calculated by using a second weightingcoefficient smaller than the first weighting coefficient for part of thepixels in the predetermined range, and wherein the calculation unit isconfigured to determine, depending on the size of the predeterminedrange, either one of the first or second evaluation value to becalculated.
 8. The image pickup apparatus according to claim 1, whereinthe evaluation value is information related to a degree of coincidencebetween the first and second image signals.
 9. The image pickupapparatus according to claim 1, wherein the evaluation value isinformation related to a focusing degree of the first and second imagesignals.
 10. The image pickup apparatus according to claim 1, whereinthe calculation unit performs three-point interpolation calculationbased on the evaluation value to calculate the defocus amount.
 11. Acontrol apparatus comprising: an input unit configured to input a firstimage signal and a second image signal output from an image pickup unitbased on an incident light beam from regions of a pupil of an opticalsystem different from each other; and a calculation unit configured tocalculate an evaluation value while relatively shifting the first andsecond image signals from pixels included in a predetermined range tocalculate a defocus amount, wherein the calculation unit is configuredto change a size of the predetermined range so that a center of a visualfield range relative to a position of a target pixel does not varydepending on a shift amount between the first and second image signals.12. A control method comprising the steps of: acquiring a first imagesignal and a second image signal output from an image pickup unit basedon an incident light beam from regions of a pupil of an optical systemdifferent from each other; and calculating an evaluation value whilerelatively shifting the first and second image signals from pixelsincluded in a predetermined range to calculate a defocus amount, whereinthe step of calculating the evaluation value includes changing a size ofthe predetermined range so that a center of a visual field rangerelative to a position of a target pixel does not vary depending on ashift amount between the first and second image signals.
 13. Anon-transitory computer-readable storage medium which stores a programcausing a computer to execute a process comprising the steps of:acquiring a first image signal and a second image signal output from animage pickup unit based on an incident light beam from regions of apupil of an optical system different from each other; and calculating anevaluation value while relatively shifting the first and second imagesignals from pixels included in a predetermined range to calculate adefocus amount, wherein the step of calculating the evaluation valueincludes changing a size of the predetermined range so that a center ofa visual field range relative to a position of a target pixel does notvary depending on a shift amount between the first and second imagesignals.